{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Non-Distributed Adversarial Evaluation\n",
    "\n",
    "Adversarial Evaluation is a key component of the robustness evaluation of machine learning models. There are different ways to evaluate the robustness of a model, efficiency of a defense mechanism or the adversarial attack itself. `advsecurenet` supports a variety of adversarial evaluation techniques. Such evaluations might be time consuming by their nature. Therefore, `advsecurenet` provides a distributed adversarial evaluation option to speed up the evaluation process. In this notebook, we will demonstrate how to use non-distributed adversarial evaluation in `advsecurenet` CLI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a Configuration File\n",
    "Due to the complexity of the adversarial evaluation process, the only way to use adversarial evaluation on CLI is to use a configuration file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Create a Configuration File\n",
    "\n",
    "As a first step, you need to have a default configuration file. You can create a configuration file by running the following command:\n",
    "    \n",
    "```sh\n",
    "advsecurenet utils configs get -c adversarial_evaluation_config.yml -s -o ./adversarial_evaluation_config.yml\n",
    "```\n",
    "\n",
    "Here, we are also providing the `-o` flag to specify the output directory where the configuration file will be saved. You can also specify the output directory by passing the `-o` flag followed by the path to the directory where you want to save the configuration file. If you don't provide the `-o` flag, the configuration file will be saved in the current working directory with the name `adversarial_evaluation_config.yml`. If the file already exists, it will be overwritten. If the output path is a directory, the configuration file will be saved in that directory with the name `adversarial_evaluation_config.yml`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving default config to ./adversarial_evaluation_config.yml\n",
      "Generated default configuration file adversarial_evaluation_config.yml!\n"
     ]
    }
   ],
   "source": [
    "!advsecurenet utils configs get -c adversarial_evaluation_config.yml -s -o ./adversarial_evaluation_config.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1m**************************************************\u001b[0m\n",
      "\u001b[31m\u001b[1mDefault configuration file for the adversarial_evaluation_config.yml:\n",
      "\u001b[0m\n",
      "\u001b[32m{\n",
      "    \"evaluation_config\": {\n",
      "        \"attack\": {\n",
      "            \"name\": null,\n",
      "            \"config\": null\n",
      "        },\n",
      "        \"evaluators\": [\n",
      "            \"attack_success_rate\"\n",
      "        ],\n",
      "        \"target_models\": [\n",
      "            null\n",
      "        ]\n",
      "    }\n",
      "}\u001b[0m\n",
      "\u001b[34m\u001b[1m**************************************************\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!advsecurenet utils configs get -c adversarial_evaluation_config.yml -p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluation configuration file might look like less complex than the other configuration files. However, it depends on external configurations such as the attack configuration file and the model configuration file. In this notebook, we will use the `FGSM` attack and as evaluators, we will use `attack_success_rate` and `transferability` evaluators. To evaluate the transferability, we need to have a second model. Therefore, we will use the `ResNet18` as the source model and the `VGG16` as the target model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Get Required Configuration Files\n",
    "\n",
    "To get the required configuration files, you can run the following commands:\n",
    "\n",
    "```sh\n",
    "advsecurenet utils configs get -c fgsm_attack_config.yml -s -o ./fgsm_attack_config.yml\n",
    "advsecurenet utils configs get -c model_config.yml -s -o ./model_config.yml\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving default config to ./fgsm_attack_config.yml\n",
      "Generated default configuration file fgsm_attack_config.yml!\n"
     ]
    }
   ],
   "source": [
    "!advsecurenet utils configs get -c fgsm_attack_config.yml -s -o ./fgsm_attack_config.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving default config to ./target_model_config.yml\n",
      "Generated default configuration file model_config.yml!\n"
     ]
    }
   ],
   "source": [
    "!advsecurenet utils configs get -c model_config.yml -s -o ./target_model_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Update the Configuration File\n",
    "For the sake of this demonstration, we will update the configuration files in place, we won't create new configuration files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Run the Adversarial Evaluation\n",
    "\n",
    "Now, we can run the adversarial evaluation by running the following command:\n",
    "\n",
    "```sh\n",
    "advsecurenet evaluate adversarial eval -c ./adversarial_evaluation_config.yml\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "\n",
      "Generating adversarial samples:   0%|\u001b[31m                  \u001b[0m| 0/4 [00:00<?, ?batch/s]\u001b[0m\u001b[A\n",
      "Generating adversarial samples:  25%|\u001b[31m██▌       \u001b[0m| 1/4 [00:00<00:02,  1.24batch/s]\u001b[0m\u001b[A\n",
      "Generating adversarial samples:  50%|\u001b[31m█████     \u001b[0m| 2/4 [00:00<00:00,  2.46batch/s]\u001b[0m\u001b[A\n",
      "Generating adversarial samples:  75%|\u001b[31m███████▌  \u001b[0m| 3/4 [00:01<00:00,  3.71batch/s]\u001b[0m\u001b[A\n",
      "Generating adversarial samples: 100%|\u001b[31m██████████\u001b[0m| 4/4 [00:01<00:00,  4.72batch/s]\u001b[0m\u001b[A\n",
      "                                                                                \u001b[A\u001b[32mAttack Success Rate: 0.0667\u001b[0m\n",
      "\u001b[32mTransferability-Vgg16: 1.0000\u001b[0m\n",
      "\u001b[32mAttack completed successfully.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!advsecurenet evaluate adversarial eval -c ./adversarial_evaluation_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have successfully run the adversarial evaluation in `advsecurenet` CLI. You can try different configurations and models to evaluate the robustness of your models, efficiency of your defense mechanisms, or the adversarial attacks themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "spe",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
